Generalizing the Kelly strategy

A recent draft by Victor Haghani and Richard Dewey [1] describes an experiment where participants were given initial wealth, a coin of known bias, and could bet a (variable) proportion of their in-game wealth on a sequence of flips of this coin. Assuming log utility and uncapped reward, the optimal strategy is to bet as per the Kelly strategy. Interestingly, the participants in general did not do so many bet a larger proportion, up to 100 % of their game assets. This note shows that such behaviour can be rational, if one takes into account the effect of extraneous wealth. The optimal solution for log utility with extraneous wealth is found, and extended to the optimal solution for a wide class of utility functions with extraneous wealth. A counterintuitive result is proved : for any continuous, concave, differentiable utility function, the optimal choice at every point depends only on the probability of reaching that point. That is, the optimal choice depends only on the number of heads encountered, regardless of the sequence. Lastly, the practical calculation of the optimal bet at every stage is made possible through use of the binomial expansion, reducing the problem size from exponential to quadratic. This makes the solution practical for games with many hundreds of steps. The author thanks Vlad Ragulin 1, for introducing the original problem, and Andy Morton 2 for motivating investigation of the general case and economic interpretation of the results. 1. [Director, US Govt Bond Trading, CGML] 2.[Global Head G10 Rates, Markets Treasury and Finance, CGML] Setup The player is given 1 unit of game wealth, and the chance to bet on f flips of a coin known to be biased with probability p > 0.5 of coming up heads. At every flip, the player may bet a proportion b ε [0,1]of their current game wealth g on heads. The player also has extraneous wealth w, which is not affected by the betting. We initially assume the player's internal reward function is log utility, i.e. they aim to maximise E[ Log[g+w] ] where g is their final game wealth. If optimizing only one step ahead, the optimal bet = (2p-1)(1+ w g ) , which reduces to the Kelly betting criterion (2p-1)if w = 0. Interestingly, this is not optimal for an n-step game. As one would expect, as g ≫ w, the game wealth dominates and the optimal bet converges to (2p-1). If g ≪ w , and the game is due to end soon, the optimal bet is 1.00, which agrees with the intuition that there is little downside to betting the entire tiny stake. But for moderately long games, the risk of losing paths that could lead to large wealth implies that the first bet must be lower than 1. For example, with p = 0.6 and w = 1000, for 25 flips the optimal first bet is approximately 0.659. Convenient notation The possible paths of the game form a complete binary tree. We number the nodes of this tree with the root = 1. Node n has children 2n (if the coin gives heads) and 2n+1(if the coin gives tails). For f flips, the final level (that is, level f+1) has 2f nodes. Let gn = the game wealth at node n bn = the bet at node n pn = the probability of reaching node n (1) For example, a game with 2 flips where the player bets 0.2 of their game wealth at each stage would give : Node n belongs to level Log2[n] +1 , that is Floor[Log2[n] +1]. At level m, pk may take m+1 distinct values, corresponding to the m+1 possibilities (0, 1, ... , m) of the number of heads in m flips. Evidently g n ⩾ 0 for all n. Also g 2 n = g n * (1+bn) and g 2 n+1 = g n * (1-bn) . This gives us the following recurrences : g n = g 2 n + g 2 n+1 2 bn = g 2 n+1 g 2 n g 2 n + g 2 n+1 This is useful : if we know the game wealths at the final level, we know for free all previous bets and wealths. (2) Final game wealths as the solution to a constrained convex optimization problem Since g 1 = 1 , and by (2) , the unweighted sum of wealths at level j equals 2j-1. We seek to maximize the total utility at the final level  k =2f 2f+1-1 pk Log[gk +w] With additional equality constraint  k=2f 2f+1-1 gk = 2 And inequality constraint gk ≥ 0 for k = 2 , 2 +1, ... , 2f+1 -1 This is an optimization problem with convex & differentiable objective , affine equality constraint, and convex inequality constraint. Therefore, the KKT conditions [2] are necessary and sufficient to find an optimum. Let x a vector with 2 entries be the desired solution of this problem (x is the vector of final wealths) Introducing Lagrange multipliers λ a vector with 2 entries for the inequality constraint , and ν (nu : a scalar) for the equality constraint, we get the standard KKT conditions xk ⩾ 0 ,  k=1 2f xk = 2 , λk ⩾ 0 , λk xk = 0 , and lastly -pk xk +w -λk +ν = 0 , for k = 1, 2, ..., 2 Usefully, the above can directly be solved for x. Eliminating λk between the last 2 conditions gives : -pk xk +w +ν xk = 0, and pk xk +w ⩽ ν Now suppose ν < pk w . Then pk xk +w ⩽ ν < pk w which can only occur if xk > 0. But then, since -pk xk +w +ν xk = 0, we must have ν = pk xk +w , which implies xk = pk ν -w Conversely suppose pk w ⩽ ν. But then xk cannot be strictly greater than 0 , because if it was, then -pk xk +w +ν xk is the product of 2 terms, both strictly greater than 0, and thus cannot be 0 . Therefore we have xk = pk ν -w if pk ν -w > 0, and 0 otherwise. i.e xk = max 0, pk ν -w (3) Substituting the expression for xk into the equality constraint, we have  k=1 2f max0, pk ν -w = 2 , (4) The LHS of which is a strictly decreasing continuous function of ν, is zero for large enough ν, and also arbitrarily large if ν is small enough. By the Intermediate Value Theorem the equation has a unique solution (which can be readily determined, e.g. by binary search). Once we have ν, we have x, the vector of optimal final wealths. We then use (2) to get the optimal bets at every node. Interestingly, we see that the value of xk depends only on pk or equivalently only on the number of heads out of f flips. This holds in the general case, as is proved shortly, which provides a route to efficient calculation of the best bet at any point. Example For 4 flips, p = 0.6 w = 20, and log utility , ν = .0041 approx and the optimum game is : Paths where the player bets all their game wealth and loses are greyed out. Generalization to other utility functions We now consider a general utility function F :R → R , reasonably be assumed to be continuous, concave & differentiable. Linear and log utility are special cases of this type. Note F ' need not be continuous, although it is continuous almost everywhere. Fis concave, therefore F ' is monotonically decreasing.Therefore : F '[w] ⩾ F '[xk +w]. F ' attains a (nonunique) max at w and a (nonunique) min at w+2f and if the strict inequality F '[w] > F '[xk +w] holds then we must have xk > 0. We now follow exactly the same proof pattern as before, with F [xk +w] and F ' [xk +w] in place of Log[xk +w] and 1 xk +w As before we seek to maximize  k =2f 2f+1-1 pk F[gk +w], subject to the constraints  k=2f 2f+1-1 gk = 2 and gk ≥ 0 for k = 2 × 2 +1, ... , 2f+1 -1 Introducing , as before, Lagrange multipliers λ a vector with 2 entries for the inequality constraint , and ν (nu : a scalar) for the equality constraint, We get the standard KKT conditions xk ⩾ 0 ,  k=1 2f xk = 2 , λk ⩾ 0 , λk xk = 0 , and lastly -pk F '[xk+w ] -λk +ν = 0 , for k = 1, 2, ..., 2 Eliminating λk between the last 2 conditions as before gives (-pk F '[xk +w] +ν ) xk = 0 and pk F '[xk +w] ⩽ ν The first term is the product of two nonzero terms. If either is strictly positive, the other is zero. As before we will use this to determine xk. Define the function H as follows : H[y] = the largest total wealth z for which F '[z] ⩾ y H is decreasing and is roughly the inverse function of the derivative of the utility : given a y, H[y] gives a total wealth where the marginal utility is as close to y as possible without dropping below y. The chart below illustrates these terms with reward function = capped log utility : w+2f pk F '[xk +w ],  